Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Split out in multiple queues #230

Draft
wants to merge 1 commit into
base: dev
Choose a base branch
from
Draft

Split out in multiple queues #230

wants to merge 1 commit into from

Conversation

pjaspers
Copy link
Contributor

I added the doc so I had something to push, and we could have a discussion here on exactly what should go where.

I think we should have 3 queues:

  • critical
  • dhis2-safe
  • default

Where critical means 'a user will need this', dhis2-safe means 'we won't hurt the dhis much'?, and 'default' means 'run this when the other two are done'.

If someone or something now enqueues 300 large invoices, we will still have delays, because the workers will pick 16 items of the default queue and we will only process the critical queue once a worker becomes free again.

If we really, really wanted to have critical always working, we would need to start two sidekiq process, I think.

So below, our workers and how I would divide them:

create_dhis2_element_for_formula_mapping_worker.rb
  Fast

create_dhis2_element_worker.rb
  Abstract (but could set queu for create_dhis2 and create_missing_dhis2)
  queue: critical

create_missing_dhis2_element_for_activity_worker.rb
  Fast

dhis2_analytics_worker.rb
  Fast (analytics is slow, but triggering it is fast)
  queues: critical

dhis2_snapshot_compactor.rb
  PORO used in worker
dhis2_snapshot_worker.rb
  Snapshot per project
  queues: critical

discard_invoicing_job_worker.rb
  Fast (cleans up orphaned tasks)

invoice_for_project_anchor_worker.rb
  queue: default

invoice_simulation_worker.rb
  queue is dhis2_safe

output_dataset_worker.rb
  queues: critical

project_coc_aoc_reference_worker.rb
  queues: critical

synchronise_deg_ds_worker.rb
  queues: critical

synchronise_groups_worker.rb
  queues: critical

update_metadata_worker.rb
  queues: critical

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mestachs mestachs Awaiting requested review from mestachs

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@pjaspers